{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lexical Information Overlap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook contains some code to process and normalize the lexical information appearing in `CodeMethod` comments and implementations \n",
    "(i.e., `CodeMethod.comment` and `CodeMethod.code`, respectively).\n",
    "\n",
    "The overall processing encompasses the following steps:\n",
    "\n",
    "- (Tokens Extraction)\n",
    "    - The textual data are chunked into tokens (thanks to `nltk`)\n",
    "- (Tokens Normalization)\n",
    "    - Most common (english) stopwords are removed, as well as Java language reserved keywords;\n",
    "    - Each non-english token is processed by the **LINSEN** algorithm;\n",
    "    - Each remaining token (a.k.a, *lexeme*) is turned into lowercase letters;\n",
    "    - Resulting tokens are finally stemmed.\n",
    "    \n",
    "Once those processing steps are completed, the `jaccard_coefficient` is computed between code and comments of each method, and all the analysis information are then stored in a `CodeLexiconInfo` model instance).\n",
    "\n",
    "\n",
    "\n",
    "### Python Version \n",
    "\n",
    "This notebook requires **Python 3**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----------------------------------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preliminaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%load preamble_directives.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import Django Model for Code Lexicon Information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from source_code_analysis.models import CodeLexiconInfo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DATA FETCHING CODE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## <a href=\"#data_analysis\">SKIP</a> this part if the Database already contains `CodeLexiconInfo` data\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analysis Code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LINSEN Normalizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from lexical_analysis import LINSENnormalizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lexical Analyzer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from lexical_analysis import LexicalAnalyzer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from source_code_analysis.models import SoftwareProject\n",
    "target_sw_project = SoftwareProject.objects.get(name__iexact='CoffeeMaker')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Use RelatedManager to get all the code methods associated to the target project\n",
    "code_methods = target_sw_project.code_methods.all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Analyzing Method 1 out of 47: CoffeeMaker\n",
      "b''\n",
      "Analyzing Method 2 out of 47: addRecipe\n",
      "b''\n",
      "Analyzing Method 3 out of 47: deleteRecipe\n",
      "b''\n",
      "Analyzing Method 4 out of 47: editRecipe\n",
      "b''\n",
      "Analyzing Method 5 out of 47: addInventory\n",
      "b''\n",
      "Analyzing Method 6 out of 47: checkInventory\n",
      "b''\n",
      "Analyzing Method 7 out of 47: makeCoffee\n",
      "b''\n",
      "Analyzing Method 8 out of 47: Inventory\n",
      "b''\n",
      "Analyzing Method 9 out of 47: getChocolate\n",
      "b''\n",
      "Analyzing Method 10 out of 47: setChocolate\n",
      "b''\n",
      "Analyzing Method 11 out of 47: addChocolate\n",
      "b''\n",
      "Analyzing Method 12 out of 47: getCoffee\n",
      "b''\n",
      "Analyzing Method 13 out of 47: setCoffee\n",
      "b''\n",
      "Analyzing Method 14 out of 47: addCoffee\n",
      "b''\n",
      "Analyzing Method 15 out of 47: getMilk\n",
      "b''\n",
      "Analyzing Method 16 out of 47: setMilk\n",
      "b''\n",
      "Analyzing Method 17 out of 47: addMilk\n",
      "b''\n",
      "Analyzing Method 18 out of 47: getSugar\n",
      "b''\n",
      "Analyzing Method 19 out of 47: setSugar\n",
      "b''\n",
      "Analyzing Method 20 out of 47: addSugar\n",
      "b''\n",
      "Analyzing Method 21 out of 47: enoughIngredients\n",
      "b''\n",
      "Analyzing Method 22 out of 47: useIngredients\n",
      "b''\n",
      "Analyzing Method 23 out of 47: mainMenu\n",
      "b''\n",
      "Analyzing Method 24 out of 47: addRecipe\n",
      "b''\n",
      "Analyzing Method 25 out of 47: deleteRecipe\n",
      "b''\n",
      "Analyzing Method 26 out of 47: editRecipe\n",
      "b''\n",
      "Analyzing Method 27 out of 47: addInventory\n",
      "b''\n",
      "Analyzing Method 28 out of 47: checkInventory\n",
      "b''\n",
      "Analyzing Method 29 out of 47: makeCoffee\n",
      "b''\n",
      "Analyzing Method 30 out of 47: inputOutput\n",
      "b''\n",
      "Analyzing Method 31 out of 47: recipeListSelection\n",
      "b''\n",
      "Analyzing Method 32 out of 47: Recipe\n",
      "b''\n",
      "Analyzing Method 33 out of 47: getAmtChocolate\n",
      "b''\n",
      "Analyzing Method 34 out of 47: setAmtChocolate\n",
      "b''\n",
      "Analyzing Method 35 out of 47: getAmtCoffee\n",
      "b''\n",
      "Analyzing Method 36 out of 47: setAmtCoffee\n",
      "b''\n",
      "Analyzing Method 37 out of 47: getAmtMilk\n",
      "b''\n",
      "Analyzing Method 38 out of 47: setAmtMilk\n",
      "b''\n",
      "Analyzing Method 39 out of 47: getAmtSugar\n",
      "b''\n",
      "Analyzing Method 40 out of 47: setAmtSugar\n",
      "b''\n",
      "Analyzing Method 41 out of 47: getName\n",
      "b''\n",
      "Analyzing Method 42 out of 47: setName\n",
      "b''\n",
      "Analyzing Method 43 out of 47: getPrice\n",
      "b''\n",
      "Analyzing Method 44 out of 47: setPrice\n",
      "b''\n",
      "Analyzing Method 45 out of 47: RecipeBook\n",
      "b''\n",
      "Analyzing Method 46 out of 47: getRecipes\n",
      "b''\n",
      "Analyzing Method 47 out of 47: deleteRecipe\n",
      "b''\n"
     ]
    }
   ],
   "source": [
    "total_methods = code_methods.count()\n",
    "coefficients = list()\n",
    "for i, method in enumerate(code_methods):\n",
    "    print('Analyzing Method {0} out of {1}: {2}'.format(i+1, total_methods, method.method_name))\n",
    "    analyzer = LexicalAnalyzer(method)\n",
    "    analyzer.analyse_textual_information()\n",
    "    coefficients.append(analyzer.code_lexical_info.jaccard_coeff)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<a name=\"data_analysis\"></a>\n",
    "# DATA ANALYSIS CODE (Statistics)\n",
    "---\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from scipy import median\n",
    "from scipy import mean\n",
    "from scipy import var, std\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<SoftwareProject: CoffeeMaker (1.0)>, <SoftwareProject: JFreeChart (0.6.0)>, <SoftwareProject: JFreeChart (0.7.1)>, <SoftwareProject: JHotDraw (7.4.1)>]\n"
     ]
    }
   ],
   "source": [
    "from source_code_analysis.models import SoftwareProject\n",
    "\n",
    "projects = list()\n",
    "projects.append(SoftwareProject.objects.get(name__iexact='CoffeeMaker', version__exact='1.0'))\n",
    "projects.append(SoftwareProject.objects.get(name__iexact='Jfreechart', version__exact='0.6.0'))\n",
    "projects.append(SoftwareProject.objects.get(name__iexact='Jfreechart', version__exact='0.7.1'))\n",
    "projects.append(SoftwareProject.objects.get(name__iexact='JHotDraw', version__exact='7.4.1'))\n",
    "\n",
    "print(projects)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Jaccard Coefficient Statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Coffeemaker (1.0) & 47 & 0.0667 & 0.429 & 0.25 & 0.254 & 0.0106 & 0.103 \\\\\n",
      "Jfreechart (0.6.0) & 486 & 0.0 & 0.857 & 0.222 & 0.233 & 0.0192 & 0.139 \\\\\n",
      "Jfreechart (0.7.1) & 624 & 0.0 & 0.857 & 0.25 & 0.262 & 0.0189 & 0.138 \\\\\n",
      "Jhotdraw (7.4.1) & 2480 & 0.0 & 1.0 & 0.182 & 0.215 & 0.0284 & 0.169 \\\\\n"
     ]
    }
   ],
   "source": [
    "for project in projects:\n",
    "    code_methods = project.code_methods.all()\n",
    "    coefficients = list()\n",
    "    for method in code_methods:\n",
    "        # Check that this method has no \"wrong_association\"\n",
    "        n_evaluations = method.agreement_evaluations.count()\n",
    "        n_eval_wrong_assocation = method.agreement_evaluations.filter(wrong_association=True).count()\n",
    "        if n_evaluations == n_eval_wrong_assocation:\n",
    "            # if **all** the evaluations for the current method mark it as a wrong_association\n",
    "            # exclude it from the statistics\n",
    "            continue\n",
    "        clexicon_info = method.lexical_info\n",
    "        coefficients.append(clexicon_info.jaccard_coeff)\n",
    "    coeff = np.array(coefficients)\n",
    "    print('{proj} ({ver}) & {total} & {min:.3} & {max:.3} & {median:.3} & {mean:.3} & {variance:.3} & {devstd:.3} \\\\\\\\'.format(\n",
    "                                                                             proj = project.name.title(), ver=project.version,\n",
    "                                                                             total=coeff.size, min=coeff.min(), max=coeff.max(),\n",
    "                                                                             median=median(coeff), mean=coeff.mean(), \n",
    "                                                                             variance=var(coeff), devstd=std(coeff)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TFIDF Statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Coffeemaker (1.0) & 47 & 0.162 & 0.818 & 0.522 & 0.494 & 0.0309 & 0.176 \\\\\n",
      "Jfreechart (0.6.0) & 486 & 0.0 & 0.954 & 0.403 & 0.403 & 0.0414 & 0.203 \\\\\n",
      "Jfreechart (0.7.1) & 624 & 0.0 & 0.958 & 0.442 & 0.447 & 0.0416 & 0.204 \\\\\n",
      "Jhotdraw (7.4.1) & 2480 & 0.0 & 1.0 & 0.41 & 0.403 & 0.0612 & 0.247 \\\\\n"
     ]
    }
   ],
   "source": [
    "# Import Scikit-Learn\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.metrics.pairwise import linear_kernel\n",
    "\n",
    "for project in projects:\n",
    "    \n",
    "    # Populate the Doc Collection\n",
    "    document_collection = list()\n",
    "    \n",
    "    # Get Methods\n",
    "    code_methods = project.code_methods.all()\n",
    "    for method in code_methods:\n",
    "        # Check that this method has no \"wrong_association\"\n",
    "        n_evaluations = method.agreement_evaluations.count()\n",
    "        n_eval_wrong_assocation = method.agreement_evaluations.filter(wrong_association=True).count()\n",
    "        if n_evaluations == n_eval_wrong_assocation:\n",
    "            # if **all** the evaluations for the current method mark it as a wrong_association\n",
    "            # exclude it from the statistics\n",
    "            continue\n",
    "        \n",
    "        clexicon_info = method.lexical_info\n",
    "        document_collection.append(clexicon_info.normalized_comment)\n",
    "        document_collection.append(clexicon_info.normalized_code)\n",
    "    \n",
    "    vectorizer = TfidfVectorizer(input='content', sublinear_tf=True, lowercase=False)\n",
    "    tfidf_values = vectorizer.fit_transform(document_collection)\n",
    "    \n",
    "    #cosine_sim_vals = list()\n",
    "    #rows, cols = tfidf_values.shape\n",
    "    #for i in range(0, rows, 2):\n",
    "    #    cosine_sim_vals.append(tfidf_values[i].dot(tfidf_values[i+1].T)[0,0])\n",
    "    #cosine_sim_vals = np.array(cosine_sim_vals)\n",
    "    comments, code = tfidf_values[::2], tfidf_values[1::2]\n",
    "    kernel_matrix = linear_kernel(comments, code)  # arrays are still L2 (length) normalized\n",
    "    cosine_sim_vals = np.diag(kernel_matrix)\n",
    "    \n",
    "    print('{proj} ({ver}) & {tot} & {min:.3} & {max:.3} & {med:.3} & {mu:.3} & {var:.3} & {sigma:.3} \\\\\\\\'.format(\n",
    "            proj=project.name.title(), ver=project.version, tot=cosine_sim_vals.size, min=cosine_sim_vals.min(), \n",
    "            max=cosine_sim_vals.max(), med=median(cosine_sim_vals), mu=cosine_sim_vals.mean(), \n",
    "            var=var(cosine_sim_vals), sigma=std(cosine_sim_vals)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (TEST) Spare Analysis on the resulting structures (IR Index) --> SKIP to <a href=\"#tf_stats\">Tf Statistics</a> \n",
    "#### Coffee Maker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Methods:  2\n",
      "Docs:  4\n"
     ]
    }
   ],
   "source": [
    "coff_maker = projects[0]\n",
    "methods = coff_maker.code_methods.all()\n",
    "methods = methods[0:2]\n",
    "docs = list()\n",
    "for method in methods:\n",
    "    lex_info = method.lexical_info\n",
    "    docs.append(lex_info.normalized_comment)\n",
    "    docs.append(lex_info.normalized_code)\n",
    "print('Methods: ', len(methods))\n",
    "print('Docs: ', len(docs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/valerio/anaconda3/envs/code_comments_analysis/lib/python3.5/site-packages/sklearn/utils/fixes.py:64: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead\n",
      "  if 'order' in inspect.getargspec(np.copy)[0]:\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "vectorizer = TfidfVectorizer(input='content', sublinear_tf=True, lowercase=False)\n",
    "X = vectorizer.fit_transform(docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['add',\n",
       " 'amount',\n",
       " 'amt',\n",
       " 'chocol',\n",
       " 'coffe',\n",
       " 'current',\n",
       " 'except',\n",
       " 'format',\n",
       " 'int',\n",
       " 'integ',\n",
       " 'inventori',\n",
       " 'must',\n",
       " 'number',\n",
       " 'paramet',\n",
       " 'pars',\n",
       " 'posit',\n",
       " 'string',\n",
       " 'unit']"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectorizer.get_feature_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n"
     ]
    }
   ],
   "source": [
    "x = X[0].toarray()\n",
    "from scipy.sparse import issparse\n",
    "print(issparse(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = x.ravel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([ 0,  1,  3,  5,  6, 10, 12, 13, 17]),)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.where(x>0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.1975104 ,  0.29840377,  0.62623383,  0.29840377,  0.1975104 ,\n",
       "         0.33441418,  0.1975104 ,  0.29840377,  0.33441418]])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.take(x, np.where(x>0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.1975104 ,  0.29840377,  0.62623383,  0.29840377,  0.1975104 ,\n",
       "        0.33441418,  0.1975104 ,  0.29840377,  0.33441418])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x[np.where(x>0)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['add', 'amount', 'amt', 'chocol', 'coffe', 'current', 'except', 'format', 'int', 'integ', 'inventori', 'must', 'number', 'paramet', 'pars', 'posit', 'string', 'unit']\n"
     ]
    }
   ],
   "source": [
    "print(vectorizer.get_feature_names())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'add number chocol unit inventori current amount chocol unit paramet chocol inventori except'"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### JHotDraw"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Methods:  2\n",
      "Docs:  4\n"
     ]
    }
   ],
   "source": [
    "jhotdraw = projects[-1]\n",
    "methods = jhotdraw.code_methods.all()\n",
    "methods = methods[0:2]\n",
    "docs = list()\n",
    "for method in methods:\n",
    "    lex_info = method.lexical_info\n",
    "    docs.append(lex_info.normalized_comment)\n",
    "    docs.append(lex_info.normalized_code)\n",
    "print('Methods: ', len(methods))\n",
    "print('Docs: ', len(docs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('creat instanc',\n",
       " 'about action applic appledirect appledirect resourc bundl util label resourc bundl util get bundl organ jhotdraw appledirect label label configur action id')"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs[0], docs[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'creat instanc'"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "methods[0].lexical_info.normalized_comment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'about action applic appledirect appledirect resourc bundl util label resourc bundl util get bundl organ jhotdraw appledirect label label configur action id'"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "methods[0].lexical_info.normalized_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "methods[0].example.target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"tf_stats\"></a>\n",
    "## TF Statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Coffeemaker (1.0) & 47 & 0.163 & 0.849 & 0.578 & 0.574 & 0.0365 & 0.191 \\\\\n",
      "Jfreechart (0.6.0) & 486 & 0.0 & 0.946 & 0.491 & 0.486 & 0.0523 & 0.229 \\\\\n",
      "Jfreechart (0.7.1) & 624 & 0.0 & 0.946 & 0.563 & 0.527 & 0.0459 & 0.214 \\\\\n",
      "Jhotdraw (7.4.1) & 2480 & 0.0 & 1.0 & 0.469 & 0.435 & 0.0663 & 0.258 \\\\\n"
     ]
    }
   ],
   "source": [
    "# Import Scikit-Learn\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "## TODO: See the following \"Optimization\" subsections to see tests\n",
    "from sklearn.metrics.pairwise import linear_kernel  # array are still L2 normalized\n",
    "\n",
    "for project in projects:\n",
    "    \n",
    "    # Get Methods\n",
    "    code_methods = project.code_methods.all()\n",
    "    \n",
    "    # Populate the Doc Collection\n",
    "    document_collection = list()\n",
    "    for method in code_methods:\n",
    "        \n",
    "        # Check that this method has no \"wrong_association\"\n",
    "        n_evaluations = method.agreement_evaluations.count()\n",
    "        n_eval_wrong_assocation = method.agreement_evaluations.filter(wrong_association=True).count()\n",
    "        if n_evaluations == n_eval_wrong_assocation:\n",
    "            # if **all** the evaluations for the current method mark it as a wrong_association\n",
    "            # exclude it from the statistics\n",
    "            continue\n",
    "        \n",
    "        clexicon_info = method.lexical_info\n",
    "        document_collection.append(clexicon_info.normalized_comment)\n",
    "        document_collection.append(clexicon_info.normalized_code)\n",
    "    \n",
    "    vectorizer = TfidfVectorizer(input='content', sublinear_tf=False, lowercase=False, use_idf=False)\n",
    "    tf_values = vectorizer.fit_transform(document_collection)\n",
    "    \n",
    "    #cosine_sim_vals = list()\n",
    "    #rows, cols = tf_values.shape\n",
    "    #for i in range(0, rows, 2):\n",
    "    #    cosine_sim_vals.append(tf_values[i].dot(tf_values[i+1].T)[0,0])\n",
    "    #cosine_sim_vals = np.array(cosine_sim_vals)\n",
    "    \n",
    "    comments, code = tf_values[::2], tf_values[1::2]\n",
    "    kernel_matrix = linear_kernel(comments, code)\n",
    "    cosine_sim_vals = np.diag(kernel_matrix)\n",
    "    \n",
    "    print('{proj} ({ver}) & {total} & {min:.3} & {max:.3} & {median:.3} & {mean:.3} & {variance:.3} & {devstd:.3} \\\\\\\\'.format(\n",
    "                                                                                 proj = project.name.title(), ver=project.version,\n",
    "                                                                                 total=cosine_sim_vals.size, \n",
    "                                                                                 min=cosine_sim_vals.min(), \n",
    "                                                                                 max=cosine_sim_vals.max(), \n",
    "                                                                                 median=median(cosine_sim_vals), \n",
    "                                                                                 mean=cosine_sim_vals.mean(), \n",
    "                                                                                 variance=var(cosine_sim_vals), \n",
    "                                                                                 devstd=std(cosine_sim_vals)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Trying to optimize the `cosine_similarity` computation replacing the `cosine_sim_vals` list\n",
    "(try using `np.vstack`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/valerio/anaconda3/envs/code_comments_analysis/lib/python3.5/site-packages/sklearn/utils/fixes.py:64: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead\n",
      "  if 'order' in inspect.getargspec(np.copy)[0]:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([ 0.60911973,  0.57123082,  0.48069991,  0.26437923,  0.60911973,\n",
       "        0.29431416,  0.17717904,  0.60911973,  0.29152788,  0.4093343 ,\n",
       "        0.26356142,  0.71350747,  0.68495483,  0.30869327,  0.71070995,\n",
       "        0.25701787,  0.37501614,  0.70250359,  0.65416068,  0.70250359,\n",
       "        0.70250359,  0.42671532,  0.37145891,  0.42671532,  0.63666839,\n",
       "        0.70924982,  0.76713779,  0.42671532,  0.20732429,  0.37453819,\n",
       "        0.40207095,  0.41140922,  0.16396579,  0.16187423,  0.42900185,\n",
       "        0.5221587 ,  0.56473598,  0.51437628,  0.56473598,  0.56473598,\n",
       "        0.62759231,  0.57385856,  0.62759231,  0.81802882,  0.58340737,\n",
       "        0.62759231,  0.33016113])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "# Target Project (as this is just an example)\n",
    "project = projects[0]\n",
    "    \n",
    "# Get Methods\n",
    "code_methods = project.code_methods.all()\n",
    "\n",
    "# Populate the Doc Collection\n",
    "document_collection = list()\n",
    "for method in code_methods:\n",
    "    clexicon_info = method.lexical_info\n",
    "    document_collection.append(clexicon_info.normalized_comment)\n",
    "    document_collection.append(clexicon_info.normalized_code)\n",
    "\n",
    "vectorizer = TfidfVectorizer(input='content', sublinear_tf=True, lowercase=False)\n",
    "tfidf_values = vectorizer.fit_transform(document_collection)\n",
    "\n",
    "rows, cols = tfidf_values.shape\n",
    "cosine_sim_vals = tfidf_values[0].dot(tfidf_values[1].T)[0,0]\n",
    "for i in range(2, rows, 2):\n",
    "    cosine_sim_vals = np.vstack((cosine_sim_vals, tfidf_values[i].dot(tfidf_values[i+1].T)[0,0]))\n",
    "\n",
    "cosine_sim_vals.ravel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.60911973,  0.57123082,  0.48069991,  0.26437923,  0.60911973,\n",
       "        0.29431416,  0.17717904,  0.60911973,  0.29152788,  0.4093343 ,\n",
       "        0.26356142,  0.71350747,  0.68495483,  0.30869327,  0.71070995,\n",
       "        0.25701787,  0.37501614,  0.70250359,  0.65416068,  0.70250359,\n",
       "        0.70250359,  0.42671532,  0.37145891,  0.42671532,  0.63666839,\n",
       "        0.70924982,  0.76713779,  0.42671532,  0.20732429,  0.37453819,\n",
       "        0.40207095,  0.41140922,  0.16396579,  0.16187423,  0.42900185,\n",
       "        0.5221587 ,  0.56473598,  0.51437628,  0.56473598,  0.56473598,\n",
       "        0.62759231,  0.57385856,  0.62759231,  0.81802882,  0.58340737,\n",
       "        0.62759231,  0.33016113])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "alt_method = np.einsum('ij,ij->i', tfidf_values[::2,].toarray(), tfidf_values[1::2,].toarray())\n",
    "alt_method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(47,)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "alt_method.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(47,)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cosine_sim_vals.ravel().shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.testing.assert_allclose(cosine_sim_vals.ravel(), alt_method)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics.pairwise import cosine_similarity, linear_kernel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(47, 104) (47, 104)\n"
     ]
    }
   ],
   "source": [
    "comments, code = tfidf_values[::2], tfidf_values[1::2]\n",
    "print(comments.shape, code.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.60911973,  0.57123082,  0.48069991,  0.26437923,  0.60911973,\n",
       "        0.29431416,  0.17717904,  0.60911973,  0.29152788,  0.4093343 ,\n",
       "        0.26356142,  0.71350747,  0.68495483,  0.30869327,  0.71070995,\n",
       "        0.25701787,  0.37501614,  0.70250359,  0.65416068,  0.70250359,\n",
       "        0.70250359,  0.42671532,  0.37145891,  0.42671532,  0.63666839,\n",
       "        0.70924982,  0.76713779,  0.42671532,  0.20732429,  0.37453819,\n",
       "        0.40207095,  0.41140922,  0.16396579,  0.16187423,  0.42900185,\n",
       "        0.5221587 ,  0.56473598,  0.51437628,  0.56473598,  0.56473598,\n",
       "        0.62759231,  0.57385856,  0.62759231,  0.81802882,  0.58340737,\n",
       "        0.62759231,  0.33016113])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kernel = linear_kernel(comments, code)\n",
    "np.diag(kernel)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from numpy.testing import assert_array_almost_equal\n",
    "assert_array_almost_equal(alt_method, np.diag(kernel))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.60911973,  0.57123082,  0.48069991,  0.26437923,  0.60911973,\n",
       "        0.29431416,  0.17717904,  0.60911973,  0.29152788,  0.4093343 ,\n",
       "        0.26356142,  0.71350747,  0.68495483,  0.30869327,  0.71070995,\n",
       "        0.25701787,  0.37501614,  0.70250359,  0.65416068,  0.70250359,\n",
       "        0.70250359,  0.42671532,  0.37145891,  0.42671532,  0.63666839,\n",
       "        0.70924982,  0.76713779,  0.42671532,  0.20732429,  0.37453819,\n",
       "        0.40207095,  0.41140922,  0.16396579,  0.16187423,  0.42900185,\n",
       "        0.5221587 ,  0.56473598,  0.51437628,  0.56473598,  0.56473598,\n",
       "        0.62759231,  0.57385856,  0.62759231,  0.81802882,  0.58340737,\n",
       "        0.62759231,  0.33016113])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "alt_method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.60911973,  0.57123082,  0.48069991,  0.26437923,  0.60911973,\n",
       "        0.29431416,  0.17717904,  0.60911973,  0.29152788,  0.4093343 ,\n",
       "        0.26356142,  0.71350747,  0.68495483,  0.30869327,  0.71070995,\n",
       "        0.25701787,  0.37501614,  0.70250359,  0.65416068,  0.70250359,\n",
       "        0.70250359,  0.42671532,  0.37145891,  0.42671532,  0.63666839,\n",
       "        0.70924982,  0.76713779,  0.42671532,  0.20732429,  0.37453819,\n",
       "        0.40207095,  0.41140922,  0.16396579,  0.16187423,  0.42900185,\n",
       "        0.5221587 ,  0.56473598,  0.51437628,  0.56473598,  0.56473598,\n",
       "        0.62759231,  0.57385856,  0.62759231,  0.81802882,  0.58340737,\n",
       "        0.62759231,  0.33016113])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cossim = cosine_similarity(comments, code)\n",
    "np.diag(cossim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "assert_array_almost_equal(alt_method, np.diag(cossim))\n",
    "assert_array_almost_equal(np.diag(cossim), np.diag(kernel))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TFIDF Statistics separated by Agreement Rates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "NC\n",
      "\n",
      "Coffeemaker (1.0) & 20 & 0.164 & 0.818 & 0.543 & 0.471 & 0.0312 & 0.177 \\\\\n",
      "Jfreechart (0.6.0) & 55 & 0.0 & 0.731 & 0.299 & 0.317 & 0.0326 & 0.181 \\\\\n",
      "Jfreechart (0.7.1) & 68 & 0.0 & 0.812 & 0.341 & 0.336 & 0.0485 & 0.22 \\\\\n",
      "Jhotdraw (7.4.1) & 1025 & 0.0 & 0.93 & 0.322 & 0.304 & 0.0515 & 0.227 \\\\\n",
      "\n",
      "DK\n",
      "\n",
      "Coffeemaker (1.0) & \\multicolumn{7}{c|}{N.A.} \\\\\n",
      "Jfreechart (0.6.0) & 24 & 0.0 & 0.656 & 0.388 & 0.368 & 0.0395 & 0.199 \\\\\n",
      "Jfreechart (0.7.1) & 36 & 0.0 & 0.844 & 0.406 & 0.405 & 0.0363 & 0.191 \\\\\n",
      "Jhotdraw (7.4.1) & 693 & 0.0 & 1.0 & 0.492 & 0.474 & 0.0671 & 0.259 \\\\\n",
      "\n",
      "CO\n",
      "\n",
      "Coffeemaker (1.0) & 27 & 0.162 & 0.767 & 0.429 & 0.511 & 0.03 & 0.173 \\\\\n",
      "Jfreechart (0.6.0) & 406 & 0.0 & 0.954 & 0.421 & 0.418 & 0.0411 & 0.203 \\\\\n",
      "Jfreechart (0.7.1) & 520 & 0.0 & 0.958 & 0.457 & 0.465 & 0.0389 & 0.197 \\\\\n",
      "Jhotdraw (7.4.1) & 760 & 0.0 & 0.971 & 0.461 & 0.47 & 0.0469 & 0.217 \\\\\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from evaluations import Judge\n",
    "\n",
    "\n",
    "judges_combinations = (('leonardo.nole', 'rossella.linsalata'),\n",
    "                       ('leonardo.nole', 'antonio.petrone'),\n",
    "                       ('leonardo.nole', 'antonio.petrone'),\n",
    "                       ('leonardo.nole', 'rossella.linsalata'),)\n",
    "\n",
    "CODES_Labels = ('NC', 'DK', 'CO')\n",
    "from collections import defaultdict\n",
    "stats_results = defaultdict(list)\n",
    "\n",
    "for pno, project in enumerate(projects):\n",
    "\n",
    "    # Get Methods\n",
    "    code_methods = project.code_methods.all()\n",
    "\n",
    "    # Populate the Doc Collection\n",
    "    document_collection = list()\n",
    "    method_ids_map = dict()  # Map (dict) to store the association method.pk --> Row index in Tfidf Matrix\n",
    "    for mno, method in enumerate(code_methods):\n",
    "        clexicon_info = method.lexical_info\n",
    "        document_collection.append(clexicon_info.normalized_comment)\n",
    "        document_collection.append(clexicon_info.normalized_code)\n",
    "        method_ids_map[method.id] = mno*2\n",
    "\n",
    "    vectorizer = TfidfVectorizer(input='content', sublinear_tf=True, lowercase=False)\n",
    "    tfidf_values = vectorizer.fit_transform(document_collection)\n",
    "\n",
    "    j1_usrname, j2_usrname = judges_combinations[pno]\n",
    "    j1 = Judge(j1_usrname, project.name, project.version)\n",
    "    j2 = Judge(j2_usrname, project.name, project.version)\n",
    "    \n",
    "    j1_evals = j1.three_codes_evaluations\n",
    "    j2_evals = j2.three_codes_evaluations\n",
    "    \n",
    "    project_stats = list()\n",
    "    for code in range(3):\n",
    "        j1_evals_code = j1_evals[code]\n",
    "        j2_evals_code = j2_evals[code]\n",
    "        \n",
    "        method_ids = j1_evals_code.intersection(j2_evals_code)\n",
    "        cosine_sim_vals = list()\n",
    "        for mid in method_ids:\n",
    "            i = method_ids_map[mid]\n",
    "            cosine_sim_vals.append(tfidf_values[i].dot(tfidf_values[i+1].T)[0,0])\n",
    "\n",
    "        cosine_sim_vals = np.array(cosine_sim_vals)\n",
    "        project_stats.append(cosine_sim_vals)\n",
    "    \n",
    "    for code in range(3):\n",
    "        vals = project_stats[code]\n",
    "        label = CODES_Labels[code]\n",
    "        if vals.size > 0:\n",
    "            stats_results[label].append('{proj} ({ver}) & {total} & {min:.3} & {max:.3} & {median:.3} & {mean:.3} & {variance:.3} & {devstd:.3} \\\\\\\\'.format(\n",
    "                                                                                 proj = project.name.title(), \n",
    "                                                                                 ver=project.version,\n",
    "                                                                                 total=vals.size, \n",
    "                                                                                 min=vals.min(), \n",
    "                                                                                 max=vals.max(), \n",
    "                                                                                 median=median(vals), \n",
    "                                                                                 mean=vals.mean(), \n",
    "                                                                                 variance=var(vals), \n",
    "                                                                                 devstd=std(vals)))\n",
    "        else:\n",
    "            stats_results[label].append('{proj} ({ver}) & \\multicolumn{{7}}{{c|}}{{N.A.}} \\\\\\\\'.format(proj = project.name.title(), \n",
    "                                                                                 ver=project.version))\n",
    "            \n",
    "for label in stats_results:\n",
    "    print('\\n{0}\\n'.format(label))\n",
    "    for value in stats_results[label]:\n",
    "        print(value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TFIDF Values Distribution (Separated by Agreement Rates)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "NC\n",
      "\n",
      "Saved Filepath: /home/valerio/Research/Code-Coherence/comments_classification/notebooks/distributions_per_rate_tfidf/NC_Coffeemaker_(1.0)_20.txt\n",
      "Saved Filepath: /home/valerio/Research/Code-Coherence/comments_classification/notebooks/distributions_per_rate_tfidf/NC_Jfreechart_(0.6.0)_55.txt\n",
      "Saved Filepath: /home/valerio/Research/Code-Coherence/comments_classification/notebooks/distributions_per_rate_tfidf/NC_Jfreechart_(0.7.1)_68.txt\n",
      "Saved Filepath: /home/valerio/Research/Code-Coherence/comments_classification/notebooks/distributions_per_rate_tfidf/NC_Jhotdraw_(7.4.1)_1025.txt\n",
      "\n",
      "DK\n",
      "\n",
      "Saved Filepath: /home/valerio/Research/Code-Coherence/comments_classification/notebooks/distributions_per_rate_tfidf/DK_Jfreechart_(0.6.0)_24.txt\n",
      "Saved Filepath: /home/valerio/Research/Code-Coherence/comments_classification/notebooks/distributions_per_rate_tfidf/DK_Jfreechart_(0.7.1)_36.txt\n",
      "Saved Filepath: /home/valerio/Research/Code-Coherence/comments_classification/notebooks/distributions_per_rate_tfidf/DK_Jhotdraw_(7.4.1)_693.txt\n",
      "\n",
      "CO\n",
      "\n",
      "Saved Filepath: /home/valerio/Research/Code-Coherence/comments_classification/notebooks/distributions_per_rate_tfidf/CO_Coffeemaker_(1.0)_27.txt\n",
      "Saved Filepath: /home/valerio/Research/Code-Coherence/comments_classification/notebooks/distributions_per_rate_tfidf/CO_Jfreechart_(0.6.0)_406.txt\n",
      "Saved Filepath: /home/valerio/Research/Code-Coherence/comments_classification/notebooks/distributions_per_rate_tfidf/CO_Jfreechart_(0.7.1)_520.txt\n",
      "Saved Filepath: /home/valerio/Research/Code-Coherence/comments_classification/notebooks/distributions_per_rate_tfidf/CO_Jhotdraw_(7.4.1)_760.txt\n"
     ]
    }
   ],
   "source": [
    "judges_combinations = (('leonardo.nole', 'rossella.linsalata'),\n",
    "                       ('leonardo.nole', 'antonio.petrone'),\n",
    "                       ('leonardo.nole', 'antonio.petrone'),\n",
    "                       ('leonardo.nole', 'rossella.linsalata'),)\n",
    "\n",
    "CODES_Labels = ('NC', 'DK', 'CO')\n",
    "from collections import defaultdict\n",
    "stats_results_paths = defaultdict(list)\n",
    "\n",
    "pwd_out = !pwd\n",
    "current_dir = pwd_out[0]\n",
    "\n",
    "folder_path = os.path.join(current_dir, 'distributions_per_rate_tfidf')\n",
    "if not os.path.exists(folder_path):\n",
    "    os.makedirs(folder_path)\n",
    "\n",
    "for pno, project in enumerate(projects):\n",
    "\n",
    "    # Get Methods\n",
    "    code_methods = project.code_methods.all()\n",
    "\n",
    "    # Populate the Doc Collection\n",
    "    document_collection = list()\n",
    "    method_ids_map = dict()  # Map (dict) to store the association method.pk --> Row index in Tfidf Matrix\n",
    "    for mno, method in enumerate(code_methods):\n",
    "        clexicon_info = method.lexical_info\n",
    "        document_collection.append(clexicon_info.normalized_comment)\n",
    "        document_collection.append(clexicon_info.normalized_code)\n",
    "        method_ids_map[method.id] = mno*2\n",
    "\n",
    "    vectorizer = TfidfVectorizer(input='content', sublinear_tf=True, lowercase=False)\n",
    "    tfidf_values = vectorizer.fit_transform(document_collection)\n",
    "\n",
    "    j1_usrname, j2_usrname = judges_combinations[pno]\n",
    "    j1 = Judge(j1_usrname, project.name, project.version)\n",
    "    j2 = Judge(j2_usrname, project.name, project.version)\n",
    "    \n",
    "    j1_evals = j1.three_codes_evaluations\n",
    "    j2_evals = j2.three_codes_evaluations\n",
    "    \n",
    "    project_stats = list()\n",
    "    for code in range(3):\n",
    "        j1_evals_code = j1_evals[code]\n",
    "        j2_evals_code = j2_evals[code]\n",
    "        \n",
    "        method_ids = j1_evals_code.intersection(j2_evals_code)\n",
    "        cosine_sim_vals = list()\n",
    "        for mid in method_ids:\n",
    "            i = method_ids_map[mid]\n",
    "            cosine_sim_vals.append(tfidf_values[i].dot(tfidf_values[i+1].T)[0,0])\n",
    "\n",
    "        cosine_sim_vals = np.array(cosine_sim_vals)\n",
    "        project_stats.append(cosine_sim_vals)\n",
    "    \n",
    "    for code in range(3):\n",
    "        vals = project_stats[code]\n",
    "        label = CODES_Labels[code]\n",
    "        if vals.size > 0:\n",
    "            filename = '{label}_{proj}_({ver})_{total}.txt'.format(label=label, \n",
    "                                                                   proj=project.name.title(), \n",
    "                                                                   ver=project.version,\n",
    "                                                                   total=vals.size)\n",
    "            filepath = os.path.join(folder_path, filename)\n",
    "            np.savetxt(filepath, vals)\n",
    "            stats_results_paths[label].append(filepath)\n",
    "            \n",
    "for label in stats_results:\n",
    "    print('\\n{0}\\n'.format(label))\n",
    "    for path in stats_results_paths[label]:\n",
    "        print('Saved Filepath:', path)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
